Shutterless far infrared (fir) camera for automotive safety and driving systems

ABSTRACT

A shutterless far-infrared (FIR) camera for advanced driver assistance systems, including at least one optical unit including at least one lens; an FIR sensor coupled to the optical unit and configured to capture FIR images; and an integrated circuit (IC) configured to process the captured FIR images to output an enhanced thermal video stream, wherein the IC further includes: a processing circuitry; and a memory containing instructions that, when executed by the processing circuitry, configure the processing circuitry to perform image corrections including at least a shutterless correction.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/515,200 filed on Jun. 5, 2017; U.S. Provisional Application No.62/526,733 filed on Jun. 29, 2017; U.S. Provisional Application No.62/543,108 filed on Aug. 9, 2017; and U.S. Provisional Application No.62/552,620 filed on Aug. 31, 2017. All of the applications referencedabove are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to automotive safety systemsand, more specifically, to shutterless far-infrared (FIR) cameras thatcan be efficiently utilized by such systems.

BACKGROUND

Thermal infrared (IR) cameras capture image wavelengths in the range ofseven and half to fourteen micrometers. A typical IR camera uses aninfrared sensor (or detector) to detect infrared energy that is guidedto the sensor through the camera's lens. IR cameras can be utilized fora variety of imaging applications including, but not limited to, passivemotion detection, night vision, thermal mapping, health care, buildinginspection, surveillance, and the like. Recently, an attempt has beenmade in the IR industry to integrate IR cameras in advanced driverassistance systems and autonomous vehicle systems.

The application, and hence the type of camera, may depend on theinfrared spectrum. The infrared spectrum lies outside the visible lightrange and consists of a near infrared section (NIR), with wavelengths of0.75-1.0 micrometers (μm); a short infrared section (SWIR) withwavelengths of 1-3.0 micrometers (μm); a mid-Infrared section (MIR),with wavelengths of 3.0-5.0 μm; and a far-infrared (FIR) section, withwavelengths of 7.0-14. μm.

One type of FIR sensor is an uncooled sensor having a small form factor.Such sensors can typically be mass-produced using low-cost technology.In a typical arrangement, an uncooled sensor does not require acryocooler for proper operation, but does require a shutter for frequentcalibration. A shutter is a mechanical element placed between the lensand the FIR sensor for alternately blocking and exposing the sensor toinfrared wavelengths. Generally, a shutter includes a flat-bladed flag,a sleeve, and an arm that connects the sleeve to the flag. The flagopens and closes at predefined time intervals.

The shutter is used during a flat-field correction (FFC) process. In aFFC process, the shutter presents a uniform temperature source to theFIR sensor. While imaging the flat-field source, the camera updates theoffset correction coefficients, resulting in a more uniform image afterthe process is completed. The duration of the FCC process is a fewhundred milliseconds, during which the image captured just prior to theshutter blocking the field of view is frozen until the FFC process ends,when the shutter is reopened. This process must occur every few minutes.

While using a shutter may improve the quality and accuracy of thethermal image captured by an FIR sensor, having a black period ofhundreds of milliseconds is not acceptable in certain applications. Forexample, using a shutter-based FIR camera in advanced driver assistancesystems and autonomous vehicle systems can pose a high risk, as thecamera must frequently shut off for the length of the FFC. In addition,shutters include moving parts that wear out over time. This may cause acamera malfunction during driving as well as shorten the life time ofthe camera.

The FIR camera designed for advanced driver assistance systems andautonomous vehicle systems should meet additional constraints other thansafety. Such constraints include a small form factor, accurate and lowlatency image processing, and low-power consumption. As such, currentlyavailable FIR cameras, and in particular shutter-based FIR cameras, arenot well adapted for such automotive applications.

It would therefore be advantageous to provide a shutter-less FIR camerafor advanced driver assistance systems and autonomous vehicles systemsthat would overcome the deficiencies noted above.

SUMMARY

A summary of several example embodiments of the disclosure follows. Thissummary is provided for the convenience of the reader to provide a basicunderstanding of such embodiments and does not wholly define the breadthof the disclosure. This summary is not an extensive overview of allcontemplated embodiments, and is intended to neither identify key orcritical elements of all embodiments nor to delineate the scope of anyor all aspects. Its sole purpose is to present some concepts of one ormore embodiments in a simplified form as a prelude to the more detaileddescription that is presented later. For convenience, the term “certainembodiments” may be used herein to refer to a single embodiment ormultiple embodiments of the disclosure.

Certain embodiments disclosed herein include a shutterless far-infrared(FIR) camera for advanced driver assistance systems, including: at leastone optical unit including at least one lens; an FIR sensor coupled tothe optical unit and configured to capture FIR images; and an integratedcircuit (IC) configured to process the captured FIR images to output anenhanced thermal video stream, wherein the IC further includes: aprocessing circuitry; and a memory containing instructions that, whenexecuted by the processing circuitry, configure the processing circuitryto perform image corrections including at least a shutterlesscorrection.

Certain embodiments disclosed herein also include an electronic circuitintegrated in a shutterless far-infrared (FIR) camera and configured toprocess FIR images, including: a processing circuitry; and a memorycontaining instructions that, when executed by the processing circuitry,configure the processing circuitry to perform image corrections,including at least a shutterless correction on the FIR images to outputan enhanced thermal video stream.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter disclosed herein is particularly pointed out anddistinctly claimed in the claims at the conclusion of the specification.The foregoing and other objects, features, and advantages of thedisclosed embodiments will be apparent from the following detaileddescription taken in conjunction with the accompanying drawings.

FIG. 1 is a block diagram of an FIR camera structured according to oneembodiment.

FIG. 2 is a block diagram of an integrated circuit (IC) integrated inthe FIR camera of FIG. 1 according to an embodiment.

FIG. 3 illustrates a pipeline of processing tasks performed by the IC ofFIG. 2 according to an embodiment.

FIGS. 4A and 4B are thermal images output by the FIR camera of FIG. 1.

FIG. 5 is an example form factor of the FIR camera designed according toembodiment.

FIG. 6 is a schematic diagram indicating various locations for mountingan FIR camera on a car.

DETAILED DESCRIPTION

It is important to note that the embodiments disclosed herein are onlyexamples of the many advantageous uses of the innovative teachingsherein. In general, statements made in the specification of the presentapplication do not necessarily limit any of the various claimedembodiments. Moreover, some statements may apply to some inventivefeatures but not to others. In general, unless otherwise indicated,singular elements may be in plural and vice versa with no loss ofgenerality. In the drawings, like numerals refer to like parts throughseveral views.

The disclosed embodiments include a far-infrared (FIR) camera that doesnot include a shutter or any other moving part that blocks the field ofview of the camera during operation. Thus, the FIR camera is ashutter-less camera. In an embodiment, the FIR camera is optimized tooperate or be integrated in advanced driver assistance systems andautonomous vehicles systems (collectively referred to hereinafter as AVSsystems).

FIG. 1 is an example block diagram of an FIR camera 100 implementedaccording to an embodiment. The FIR camera 100 includes an optical unit110 and an FIR sensor 120 coupled to an integrated circuit (IC) 130. Theoutput of the FIR camera 100 is a video stream of thermal images(thermal video stream) captured by the FIR sensor 120 and processed bythe IC 130.

In an embodiment, two thermal video streams are output by the FIR camera100. A first stream 104 includes thermal images captured by the FIRsensor 120 and processed by the IC 130 for the shutterless correctionand image enhancement. In an example embodiment, the first thermal videostream 104 is an 8-bit grayscale video stream. The first thermal videostream 104 is fed to a display 140 which may be a screen or a dashboardof a vehicle. In a further embodiment, multiple thermal video streamsare output by the FIR camera. The output streams may include resultsfrom images received from non-FIR sensors, e.g., visible wavelength, CCDor CMOS sensors.

A second stream 105 includes thermal images captured by the FIR sensor120 and processed by the IC 130 for at least shutterless correction. Inan example embodiment, the second thermal video stream 105 is a 14-bitdata stream. The second stream 105 is fed to a computer 150 for computervision processing. Specifically, the computer 150 may be configured toexecute various computer vision algorithms related to AVS and otherautonomous systems and applications. The algorithms may include, but arenot limited to, detection of objects, obstacles, or hazards on a road.For example, the processed thermal video stream as output by the FIRcamera 100 can be utilized for the detection of vehicles, pedestrians,animals, two-wheelers, black-ice spots, litter, debris, potholes, gullycovers, curbs and so on. In an embodiment, the computer 150 may be anonboard vehicle computer or a dedicated computer installed in thevehicle (not shown). In some example embodiments, the computer may be avirtual or physical computing machine operable in a cloud computingplatform (not shown).

As will be discussed below, according to some embodiments, theprocessing performed by the IC 130 is configured to enhance the qualityof the captured thermal images to allow for accurate and fast detectionof objects, obstacles and hazards on the road. The processing by the IC130 ensures passive detection of objects during day and night lightingconditions and at all weather conditions. To this end, the IC 130 isconfigured to perform one or more image processing tasks. Such tasksinclude, but are not limited to, shutterless correction, sunburnprotection, bad pixel replacement, near field correction, temporaldenoising, spatial denoising, edge enhancement, automatic gain control,local contrast, enhancement, and brightness and polarity adjustment. Inan embodiment, these tasks are performed in a pipeline manner where someor all the tasks may be utilized to process a FIR image (frame), thusensuring low latency processing.

An important image processing task performed by the IC 130 is theshutterless correction of FIR images captured by the FIR sensor 120. Asdemonstrated in FIG. 1, the FIR camera 100 does not include any shutter(or any moving part that can be viewed as shutter). The shutterlesscorrection executed by the IC 130 allows the performance of a flat-fieldcorrection without any shutter. That is, shutterless correction allowsfor a uniform FIR image.

It should be appreciated that the shutterless FIR camera 100 ismechanically reliable and meets safety standards required for AVSsystems. Further, not using a shutter allows designing an FIR camera 100with a small form factor as there is no need to include moving parts inthe design. An example form factor of the FIR camera 100 is shown inFIG. 5.

In an embodiment, the optical unit 110 includes one or more lenselements, each of which having a predetermined field of view (FOV). Inan embodiment, the lens elements are chalcogenide. The lens may beprotected by a protective window (not show in FIG. 1). Further, toprevent icing of the optical unit 110, it may be covered by a heatingmeans (not shown in FIG. 1). The heating means may include a wire or acoil.

In an embodiment, the FIR camera 100 may include multiple optical units110. Each optical unit 110 may be used independently or may be used inconjunction with one or more other optical units 110. For example, asingle FIR sensor 120 may be connected to two or more optical units 110.Further, in an embodiment, multiple FIR sensors 120 may be connected toa single IC 130. The multiple optical units 110 and/or multiple FIRsensors 120 may be used in tandem to capture and create a resultingimage with a higher final resolution than a single optical unit 110 orFIR sensor 120 may otherwise provide. As a non-limiting example, two VGA(640 pixels×480 pixels) sized FIR sensors 120 may be used to capture asingle 1280 pixels×480 pixels FIR image. The IC 130 may be configured toautomatically stitch two adjacent images together and may apply all ofthe corrections discussed herein to the final composite image.Alternatively, each individual image may be first corrected, and thenstitched together. In a further embodiment, the images are not stitched,but rather concatenated horizontally or vertically when output to adisplay or to a computer. In yet a further embodiment, the multipleoptical units 110 or FIR sensors 120 may be used for stereo visionapplications, or to provide multiple view angles of a scene.

In yet a further embodiment, multiple image streams are received andprocessed by the FIR camera 100. Such embodiments may include creatingan output stream that is a fusion of the multiple video streams,including different imaging modalities e.g., a fusion of FIR images witha visible-spectrum image, a fusion of FIR images having various focallengths, and the like. Any number of input streams may be received andprocessed to produce the fused output stream.

The FIR sensor 120 is an uncooled FIR sensor. That is, the sensor 120operates in the FIR spectrum with a wavelength of 7.0-14.0 μm. In anexample, the FIR sensor 120 is coupled through a first bus 101 to the IC130 to input the captured FIR images and metadata. In an embodiment, ajunction temperature sensor (temp sensor) 121 is included in anenclosure of the FIR sensor 120 to measure the junction temperaturetherein.

A control bus 102 is also connected between the FIR sensor 120 and theIC 130. On the bus 102, signals related to status of the sensor 120,clock, synchronization, and other digital control signals aretransferred. In an embodiment, the bus 102 may carry analog signalsindicating, for example, the current ambient temperature at the FIRsensor 120. Alternatively, the analog signal may not be part of thecontrol bus 102.

The IC 130 includes a memory, a processing circuitry, and variouscircuits and modules allowing the execution of the tasks noted herein. Adetailed block diagram of the IC 130 is provided in FIG. 2. The IC 130may be realized as a chipset, a system on a chip (SoC), a fieldprogrammable gate array (FPGA), a programmable logic device (PLD), anapplication specific integrated circuit (ASIC) or any other type ofdigital and/or analog hardware components.

The IC 130, and hence the FIR camera 100, operates in four differentmodes: calibration, a power-on-test, a function, and a standby. Thecalibration mode is performed at a lab where the IC 130 executes acalibration process based on the previous calibration points. During theprocess, the FIR camera 100 is stabilized at a predefined temperatureand both the FPGA and ambient temperatures are periodically read fromthe FIR sensor 120 to determine temperature stability.

In the power-on-test mode, the FIR camera 100 is checked, during itspower-up, if the various circuits of the IC 130 operate properly. Thisincludes, for example, performing internal self-tests for memories,logic monitors, digital and analog components. In the functional mode,the IC 130 performs image processing tasks and communicates withexternal systems and components (e.g., sensor and computer). In thestandby mode, no image processing is performed but only communicationwith the external systems and components. The standby mode may beactivated when the vehicle is not in motion.

The FIR sensor 120 and IC 130 are encapsulated in a thermal core. Thethermal core is utilized to ensure a uniform temperature for the FIRcamera 100. The temperature calibration of the thermal core is performedduring the calibration mode. The optical unit 110 is typically assembledin the FIR camera 100 after the FIR sensor 120 and IC 130 areencapsulated in the thermal core.

FIG. 2 show an example block diagram of the IC 130 designed according toone embodiment. The IC 130 includes a processing circuitry 210, ashutterless correction processor (SLCP) 220 and an image enhancingprocessor (IEP) 230 that are configured to execute the various imageprocessing tasks discussed herein. The IC 130 further includes a memory240, and a temperature sensor 260.

The IC 130 interfaces with the external elements, such as the computer140 and display 150 through a multimedia link 202. In an exampleembodiment, the media link is a gigabit multimedia serial link (GMSL).As noted above, in one embodiment, the IC 130 can be configured tooutput two thermal video streams of FIR images: a first stream (104) isprocessed by the SLCP 220 and the IEP 230 and input to the display 140,and second stream (105) is processed by the SLCP 220 and input to thedisplay 150. Both thermal video streams are output via the multimedialink 202. In a further embodiment, a single thermal video stream isoutput to the computer 150. For example, certain autonomous vehiclesystems may not include a display, and thus only require input from thethermal video stream to a computer 150.

In some optional embodiments, the IC 130 does not include the IEP 230.In such embodiments, shutterless correction is performed by the SLCP220, and the output is fed to the computer 140. A configuration of theFIR camera 100 without the IEP 130 can be utilized in autonomous vehiclesystems where a display may not be required.

In an embodiment, the memory 240 is configured to store calibrationtables. The calibration tables include at least various calibrationvalues for each pixel computed at the lab. The calibration values mayinclude the Gain and Offset calculated from two temperature points (T₁,T₂) for the purpose overcoming the irregularities in the FIR sensor andunifying the pixels' response to IR radiation for a wide range ofambient temperatures. The calibration table also includes a drift valuedetermined for each pixel at each temperature point during a calibrationprocess. In an embodiment, the tables also store various parameters'values to set the FIR camera 100.

The memory 240 may further store computer readable instructions to beexecuted by the processing circuitry 210, the SLCP 220 and the IEP 230.The computer readable instructions shall be construed broadly to meanany type of instructions, whether referred to as software, firmware,middleware, microcode, hardware description language, or otherwise.Instructions may include code (e.g., in source code format, binary codeformat, executable code format, or any other suitable format of code).The instructions, when executed by the processing circuitry 210, theSLCP 220 and/or IEP 230, cause these processors to perform the variousembodiments and tasks described herein. The memory 240 may be volatile(e.g., RAM, etc.), non-volatile (e.g., ROM, flash memory, etc.), or acombination thereof.

In an optional configuration, the camera IC 130 includes a motiondetection unit (MDU) 250. The MDU 250 includes one or moreaccelerometers utilized to determine if the FIR camera 100 (and hencethe vehicle) is in motion. When the MDU 250 is not utilized, a motiondetection is based on the captured imaged. A motion indication isprovided to the SLCP 220 and IEP 230. The temp sensor 260 measures theambient temperature inside the enclosure of the FIR camera 100.

The processing circuitry 210 is configured to control the variouselements of the IC 130. The circuitry 210 is further configured topower-up the FIR sensor 120, upload the initial calibration tables andset the FIR sensor 120 and the SLCP 220 and IEP 230 based on theparameters' values in such tables. In an embodiment, the circuitry 210periodically samples the temperature as measured by the temp sensor 260,and uploads a set of calibration tables from the memory 240 that includecalibration values for the measured temperature range. To allow for fastexecution, some tables can be uploaded in the background, i.e., duringthe processing of FIR images.

In an embodiment, the SLCP 220 is configured to perform at leastshutterless pixel-based and scene-based corrections. The pixel-basedcorrection corrects offset, gain and drift for each pixel. In anembodiment, the offset and gain of a pixel are corrected based on therespective values in the calibration table matching a respectivetemperature as measured by the temp sensor 121 (FIG. 1). This correctioncompensates for the irregularities in the sensor and unify the pixels'response to IR radiation for a wide range of ambient temperatures. Thedrift correction includes adding drift correction to each of the pixels.Specifically, correction for each pixel is a subtraction of a product ofpre-defined drift value provided from a loaded calibration tableseparately for each pixel and a gamma value which is calculated from theframes produced by the FIR sensor. The pre-defined drift value isselected from table based on the current measured temperature at the FIRsensor 121.

According to some embodiments, the drift correction is based on atemperature drift correction for each of the pixels in the images outputthe FIR sensor. This may be necessary if the temperature values of theFIR camera as a whole, and the FIR sensor and IC individually, shiftduring operation.

The drift correction may be based on estimating a drift coefficient γusing a drift pattern recorded during the calibration stage of the FIRcamera 100. In an embodiment, estimation of the drift coefficient γ isperformed iteratively with small incremental updates based on each inputimage, so that any residual noise or new noise, e.g., due to ambienttemperature shifts, is detected and removed with each subsequent image.

The ambient drift of an input image is corrected using the estimateddrift coefficient γ. A drift corrected image (I_(out)) can berepresented as follows:

$\begin{matrix}{I_{out} = \frac{I_{in} - {\gamma \cdot D} - O}{G}} & {{Eq}.\mspace{14mu} 1}\end{matrix}$

where, is I_(in) is the input image (produced by the FIR sensor 120); γis the drift coefficient, which is the same for all pixels; ‘D’ is thedrift pattern, unique per pixel; ‘O’ is an offset gain pattern; and ‘G’is the scene gain pattern value, unique per pixel, where D, O and Gvalues are determined during initial calibration of the FIR sensor 120.

In some configurations, the gain and offset are corrected prior to thedrift correction, in which case such values are not required to estimatethe drift coefficient γ. Thus, a more generalized version of the driftcorrection calculation uses a 2-points corrected input image (I_(2pt)),drift coefficient and the drift pattern from the calibration, and can berepresented as the following equation:

I _(out) =I _(2pt) ·γ·{circumflex over (D)}  Eq. 2

In Equation 2, {circumflex over (D)} is the value of the drift pattern(D) divided by the gain pattern (G).

The drift coefficient γ used for the correction is derived from thedrift pattern from the initial calibration and the resulting pattern ofdrift within an input image after a high pass filter has been appliedthereto.

A generalized equation for the drift coefficient is the quotient of theproduct of the drift pattern and the input image pixel values afterapplying a high pass filter thereto, and the product of the driftpattern and the drift pattern after applying a high pass filter thereto,represented by the following equation:

$\begin{matrix}{\gamma = \frac{\langle{{H_{1}(D)},{H_{2}\left( I_{in} \right)}}\rangle}{\langle{{H_{1}(D)},{H_{2}(D)}}\rangle}} & {{Eq}.\mspace{14mu} 3}\end{matrix}$

where, H₁, H₂ are pre-processing operators designed to minimizecorrelation between clean image of the scene and the drift pattern.Without loss of generality, these operators may include high passfiltering and masking. An example of such operations, without loss ofgenerality, can be H₁=identity map and H₂=a high-pass filter.

In Equation 3 (Eq. 3), γ is a scalar value obtained as a ratio of 2inner products. Therefore, it is one value for all pixels at this stage.In an embodiment, the scalar value can be converted to a (slowlyvarying) value per pixel by using Eq. 3 on different segments of theimage and interpolating results. An example for such embodiment isdiscussed below.

In an embodiment, the scene-based correction includes removing fixednoise patterns. Such a pattern resulted from a high frequency signalremaining in the processed FIR image after subtracting a low-passsmoothed image from the original image. In a shutterless implementation,the resulted scene is homogeneous which leads to edges (high-frequencysignals) from the actual scene to appear in the obtained noise pattern.This may lead to real edges from the scene being mistakenly classifiedas noise. To this end, the noise patterns are determined based on imagesamples in time, and from homogeneous areas in space.

As will be discussed in detailed below, the SLCP 220 implements aplurality of modules to execute shutterless corrections and other imageenhancing tasks. In an embodiment, the scene-based correction alsoincludes correcting pixels that suffer from “sunburn effect.” Such aneffect is caused by exposing the FIR camera to direct sun which bringsthe exposed pixels into deep saturation due to overexposure. The outcomeof this effect is characterized by a ghost footprint of sun similar to afixed pattern noise which may last for minutes or weeks, depending onthe exposure time. According to the disclosed embodiment, the sunburncorrection also removes ghost footprints from the image in a few seconds(e.g., 2-5 seconds).

At minimum, the SLCP 220 is implemented to perform pixel-basedshutterless correction, column noise suppression, scene-based noiseremoval, and near field correction. In an embodiment, the SLCP 220 canbe configured to perform additional image processing tasks to provide ahigh-quality thermal video stream that would allow for the accuratedetection of objects. The image processing tasks are performed in apipeline manner by the various modules as described as illustrated inFIG. 3.

The bad pixel correction (BPC) module 301 masks pixels designed in a badpixel map. Such pixels do not meet the basic quality requirements asrequired by a manufacture of the FIR sensor. The bad pixels in the mapcan be identified by the manufacture or during a test performed on theFIR sensor at the lab. The bad pixel map is saved in the memory 240.

The sunburn detection module 302 detects pixels that suffer fromsunburn, i.e., pixels that exposed to deep saturation. In an embodiment,the received image from an FIR sensor is analyzed by computing thecaptured resistive value associated with each pixel of the image anddetermining if the value of a pixel exceeds a predetermined sunburnthreshold value (PSTV) during a predefined time period. The PSTV is avalue that represents an oversaturation of a pixel that is likely to bea sunburn artifact caused by a bright light source, such as the sun orthe high beams of a car. The determination can be made using thefollowing equation:

Max>P>PSTV  Eq. 4

where Max represents the maximum possible value for a pixel, Prepresents the current value of the pixel, and PSTV is the predeterminedthreshold value. If P falls between the PSTV and the Max value for apredefined time period, it is assigned a sunburn designation.

It is determined if the neighboring pixels of a current pixel exceed thePSTV. In a non-limiting example, an r by r pixel square around the pixelis analyzed (where r is an integer gather than 1). Even if the pixelitself is equal to or less than the PSTV, if any of the r by rneighboring pixels exceeds the PSTV and lower than the max value over apredetermined period of time, the current pixel is assigned a sunburndesignation. This ensures that a single pixel positioned within a largerarea that is determined to be an artifact caused by a sunburn is stilltreated as a sunburn pixel. This method is effective in identifyingsunburn pixels, as sunburn artifacts appear as localized patterns with ahigh correlation with neighboring pixels. Thus, the value of aneighboring pixel, when measured over a sufficient time-period, mayindicate the presence of a sunburn artifact. This is in contrast toother unwanted artifacts introduced into images, such as fixed patternnoise, which appears at random positioning within an image and has a lowcorrelation with neighboring pixels.

In an embodiment, if P=Max, it is determined that the evaluated pixel isbeing currently exposed to a bright source of light. Because the brightlight is determined to be present within the image, there is no need forinstant correction, as the scene is being properly displayed. Rather,the pixel's value is tracked and when its value exceeds the PSTV for thepredetermined period of time, the correction is triggered.

If none of the neighboring pixels are determined to exceed the PSTV forthe predetermined period of time, the pixel is assigned a non-sunburndesignation. The process continues until the entire image has beenevaluated. It may be determined if there are additional pixels withinthe pixel matrix to be analyzed.

The pixel-based shutterless correction (PBSC) module 303 performscorrection of the offset, gain and drift value for each pixel asdiscussed above. The column noise suppression (CNS) module 304suppresses fixed pattern column noise in the FIR image capture by theFIR sensor. The bad pixel correction (BPC) module 305 attempts tocorrect bad pixels by approximating each bad pixel using an optimalinterpolation based on the pixel predefined number (e.g., 8) of itsclosest neighboring pixels.

The near field correction (NFC) 306 corrects undesired patternssuperimposed on the scene image by various physical effects such asreflections of the camera case and lens. Such patterns are modeledduring the calibration mode. During the operational mode, the NFC module306 identifies a best fit between the pattern and the input image andsubtracts the pattern from the image accordingly. The NFC module 306handle two such patterns either independently or as a linearcombination. The required pattern images and parameters are provided bythe processing circuitry 210 at power up and updated when necessary.

The scene-based noise removal (SBNR) module 307 performs the shutterlessscene-based and sunburn correction tasks discussed above. The module 307is required for a proper shutterless operation.

To perform the scene-based nonuniformity correction, the SLCP 220 isconfigured to detect smooth regions to learn or otherwise estimate thenoise patterns, detect if the FIR camera (hence the vehicle) is inmotion (since the noise estimation is performed during motion only), anddetect regions that undergo changes, as some regions will not changefrom frame to frame even during motion. According to some disclosedembodiments, the residual nonuniformity estimate within an image at aspecific time (D_(r)) can be represented as follows:

D=D _(t−1)+(1−α)·D _(curr)  Eq.5

where, is D_(t) is the noise estimated at time t, D_(t−1) is the noisepreviously estimated at time t−1, and D_(curr) is the noise estimatedfrom the currently captured image, and α is a learning rate having ascalar value from 0 to 1. At time t=0, the value is D_(t) is 0. Itshould be noted that D_(t), D_(t−1), and D_(curr), are matrices havingthe size ‘m’ by ‘n’, for an image containing ‘m’ by ‘n’ pixels. In anembodiment, the noise of each pixel within the image is calculated, andthus each pixel is assigned a noise value.

In an example embodiment, the value D_(curr) can be represented asfollows:

D _(curr) =HI _(d) ·M,  Eq. 6

I _(d) =I−D _(t−1)  Eq. 7

where HI_(d) represents a high frequency representation of the image.HI_(d) can be determined by applying high-pass filter on the diffidenceimage I_(d) between current input image I and the estimated noiseD_(t−1) at a time t−1. In an embodiment, I_(h)=HI_(d) is calculated byapplying a high-pass filter on I_(d). The computation of I_(d) isperformed for every pixel in the image. In an embodiment, bad andsaturated pixels are excluded from the computation by including them inthe mask M. Here, the matrix M represents a masking, namely pixels inthe image that are designated to be included in the calculation of theresidual nonuniformity present in the current image. In an exampleembodiment, the mask matrix M is based on a combination of fourdifferent masks. In such an embodiment, M can be represented as follows:

M=M _(sat) ·M _(b) ·M _(edge) ·M _(t)  Eq. 8

where M_(sat) is a mask without over-saturated pixels, M_(b) is a maskwith pixels that are deemed to be “bad pixels” excluded,” M_(edge) is amask with edges excluded, and M_(t) is a mask with regions lackingtemporal changes excluded. All, of M_(sat), M_(b), M_(edge), M_(t) arematrices. In an embodiment, M is a binary mask with 1 denoting includedpixels and 0 denoting excluded pixels.

M_(sat) and M_(b), representing oversaturated and ‘bad’ pixels that havebeen identified by analyzing the image, are removed from the imageregions used to calculate the scene-based nonuniformity correction. ‘Badpixels’ are pixels within the image that are known a-priori to bedefective. These pixels are removed from the calculation by applying amask M_(b) on the image. In contrast, M_(edge) and M_(t) exclude “live”pixels that must be dropped because they either come from regions thathave too strong edges, or are not dynamical enough.

The matrix M_(edge) is calculated by comparing a pixel to neighboringpixels, for example creating a q×q (where q an integer greater than 1)matrix around the pixel, and determining if there is a significant gapin pixel value.

The matrix M_(t) represents regions with changes between two images. Forexample, if a low contrast depiction of a distant mountain in a firstimage stays stationary relative to the second image, the correspondingpixels are masked to exclude them from the calculation, as they do notrepresent changes within the image. However, if a moving object withinthe foreground of the frame changes position between the first and thesecond image, those pixels are not masked, allowing for the calculationof noise from the changes within the image.

The h-flip module 308 implements horizontal flip of the received sensorimage. In an embodiment, the module 308 is further configured to clipthe pixels' values between minimum and maximum values. These values arepre-configured.

The time denoise module 309 is configured to perform time denoisingprocess of infrared video. The video obtained from the FIR sensorcontains temporal noise of varying types, such as white noise, salt andpepper noise (occasional flickering) and row/columns noise. In anembodiment, the module 309 is realized by an IIR filter with an adaptivethreshold, and an anti-flickering mechanism. The time denoise module 309compares changes in pixel values from frame to frame to the estimatednoise variance, to decide whether such values are caused by noise, or bysome actual change in the scenery. Based on the comparison, a signal tonoise ratio is determined.

The spatial denoise module 310 is configured to denoise spatial noise.Such noise is caused due to internal electronic noise, errors in thepixel gains and offsets and drifts caused by temperature fluctuations.In an embodiment, the module 310 replaces each noisy pixel by weightedaverage of neighboring pixels. In an embodiment, only pixels having agray level that is closer to a predefined gray level threshold arereplaced.

It should be noted that in certain configurations only some of themodules described are required as part of the processing. The variousparameters, maps, and calibration values are not required to operate thevarious modules and processes stored in the memory 240 and are notillustrated herein merely for the sake of simplicity.

In an example embodiment, the output of the spatial denoise module 310is a 14-bit grayscale video stream fed to the computer 140 (FIG. 1) andthe IEP 230.

Returning to FIG. 2, the IEP 230 processes the 14-bit stream to producean enhanced thermal video stream that can be seen by a human. In anembodiment, the IEP 230 converts the 14-bit stream to a high quality8-bit stream that can be efficiently displayed on a screen.

In an embodiment, the IEP 230 includes the modules, shown in FIG. 3,that perform image enhancement processes. The first module is abilateral filter 311 utilized to edge preserving and noise reductionfilter. The automatic gain control (AGC) module 312 is configured toreduce the influence of empty bins from the gray level range. That is,the module 312 reduces the distance between the lowest and the highestgray levels in the image to a minimum without essentially losinginformation. The dynamic range reduction allows to stretch the imagehistogram as much as possible.

The local contrast enhancement (LCE) module 313 performs a process forcontrast limited adaptive histogram equalization (CLAHE). In imageprocessing, contrast limited adaptive histogram equalization is atechnique used to improve the local contrast of an image. The adaptivehistogram equalization is performed to find the mapping for each pixelbased on its local (neighborhood) grayscale distribution.

In an embodiment, the LCE module 313 is configured to divide the imageinto overlapping blocks to reduce even more the blocking effect. In somecases, when grayscale distribution is highly localized, it may not bedesirable to transform very low-contrast images by full histogramequalization. In such cases, the mapping curve may include segments withhigh slopes, meaning that two very close grayscales might be mapped tosignificantly different grayscales. This is resolved by limiting thecontrast that is allowed through histogram equalization.

The last image processing enhancement module is thepolarity-and-brightness (PB) module 314. In an embodiment, the module314 changes between white-hot and black-hot pixels and adjust thebrightness of the image's pixels by a predefined offset.

It should be noted that the processing of the various modules shown inFIG. 3 is performed on infrared images input by the FIR sensors 120. Theresults images are still infrared images (referred herein as “FIRimages”). Thus, the processing does not change the nature of the image(e.g., from an infrared to a visible spectrum image), but ratherimproves the quality of the infrared image.

Referring again to FIG. 2, in an embodiment, the processing circuitry210 can be a single core or a multiple-core CPU. Each of the SLCP 220and IEP 230 may be realized as one or more hardware logic components andcircuits. For example, and without limitation, illustrative types ofhardware logic components that can be used include a FPGA, an ASIC, anASSP, a SOCs, a general-purpose microprocessor, a microcontroller, aDSP, a GPU, and the like, or any other hardware logic components thatcan perform calculations or other manipulations of information.

As noted above, the output thermal video stream can be processed andanalyzed to detect objects, obstacles and hazards in a scene. Followingare some examples:

FIG. 4A shows an example thermal image 400 captured at night, output bythe FIR camera 100 installed on a car (not show). The analysis of theimage 410 allows for the detection of an animal (410) crossing the road.

FIG. 4B shows an example thermal image 420 captured on a rainy night,output by the FIR camera 100 installed on a car (not show). The analysisof the FIR images allows the detection of pedestrians (430) walking onthe road.

FIG. 5 shows an example embodiment of the shutterless FIR camera 100. Inan embodiment, the shutterless FIR camera 100 is shaped a cylinderhaving a size comparable to a ‘C’ size battery. In an exampleembodiment, the diameter of the camera is 25.4 mm and the length of thecamera without the lens is 12 mm. The small size form factor of the FIRcamera 100 allows for installation of the camera on variousinconspicuous locations on a vehicle that would not change the overallappearance of the vehicle.

FIG. 6 is a schematic diagram indicating various locations for mountingan FIR camera on a car. In an example embodiment, the shutterless FIRcamera 100 can be mounted to either or both of the side mirrors 610 of avehicle 600, a grill guard (not shown), a front bumper 620, a backbumper 630, the top of the car 640, the hood 650, headlights 660, andantenna (e.g., a radio satellite antenna, not shown). Other locationsthat provide unblocked field of views are also applicable.

In an embodiment, multiple shutterless FIR cameras 100 can be installedor mounted on a single vehicle. In such a configuration, the wide fieldof view of the multiple cameras are responsible of crossing objectsdetection and scene understanding while the narrow field of view camerahandles far range objects (or obstacles).

In yet another embodiment, the vehicle may be equipped with otherpassive or active detection means, such as CMOS cameras, LIDAR andRADAR. In such configurations, detection of objects, obstacles orhazards may be performed based on data captured by such detection meansand FIR images captured and processed by the disclosed shutterless FIRcameras 100.

It should be noted that the disclosed shutterless FIR camera can bedisclosed on any type of vehicle. For example, the disclosed FIR cameracan be mounted, installed or integrated in cars, buses, motorcycles,tracks, trains, drones, unmanned aerial vehicles, airplanes, airballoons, armored vehicles, agricultural machines, tanks, and so on. Thevehicle may be an autonomous vehicle or operated by an operator (e.g., adriver).

Some embodiments disclosed herein can be implemented as hardware,firmware, software, or any combination thereof. Moreover, the softwareis preferably implemented as an application program tangibly embodied ona program storage unit or computer readable medium consisting of parts,or of certain devices and/or a combination of devices. The applicationprogram may be uploaded to, and executed by, a machine comprising anysuitable architecture. Preferably, the machine is implemented on acomputer platform having hardware such as one or more central processingunits (“CPUs”), a memory, and input/output interfaces. The computerplatform may also include an operating system and microinstruction code.The various processes and functions described herein may be either partof the microinstruction code or part of the application program, or anycombination thereof, which may be executed by a CPU, whether or not sucha computer or processor is explicitly shown. In addition, various otherperipheral units may be connected to the computer platform such as anadditional data storage unit and a printing unit. Furthermore, anon-transitory computer readable medium is any computer readable mediumexcept for a transitory propagating signal.

As used herein, the phrase “at least one of” followed by a listing ofitems means that any of the listed items can be utilized individually,or any combination of two or more of the listed items can be utilized.For example, if a system is described as including “at least one of A,B, and C,” the system can include A alone; B alone; C alone; A and B incombination; B and C in combination; A and C in combination; or A, B,and C in combination.

It should be understood that any reference to an element herein using adesignation such as “first,” “second,” and so forth does not generallylimit the quantity or order of those elements. Rather, thesedesignations are generally used herein as a convenient method ofdistinguishing between two or more elements or instances of an element.Thus, a reference to first and second elements does not mean that onlytwo elements may be employed there or that the first element mustprecede the second element in some manner. Also, unless statedotherwise, a set of elements comprises one or more elements.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the principlesof the disclosed embodiment and the concepts contributed by the inventorto furthering the art, and are to be construed as being withoutlimitation to such specifically recited examples and conditions.Moreover, all statements herein reciting principles, aspects, andembodiments of the disclosed embodiments, as well as specific examplesthereof, are intended to encompass both structural and functionalequivalents thereof. Additionally, it is intended that such equivalentsinclude both currently known equivalents as well as equivalentsdeveloped in the future, i.e., any elements developed that perform thesame function, regardless of structure.

What is claimed is:
 1. A shutterless far-infrared (FIR) camera foradvanced driver assistance systems, comprising: at least one opticalunit including at least one lens; an FIR sensor coupled to the opticalunit and configured to capture FIR images; and an integrated circuit(IC) configured to process the captured FIR images to output an enhancedthermal video stream, wherein the IC further comprises: a processingcircuitry; and a memory containing instructions that, when executed bythe processing circuitry, configure the processing circuitry to: performimage corrections including at least a shutterless correction.
 2. Theshutterless FIR camera of claim 1, wherein the IC is further configuredto output a first output thermal video stream to a computer.
 3. Theshutterless FIR camera of claim 2, wherein the IC is further configuredto output a second output thermal video stream to a display.
 4. Theshutterless FIR camera of claim 1, wherein the FIR camera furthercomprises: a plurality of image sensors coupled to the IC.
 5. Theshutterless FIR camera of claim 4, wherein at least one of the pluralityof image sensors is an FIR sensor.
 6. The shutterless FIR camera ofclaim 4, wherein the IC is further configured to output a thermal videostream that is an image fusion of the plurality of image sensors.
 7. Theshutterless FIR camera of claim 1, wherein the FIR sensor and the IC areencapsulated within a thermal core.
 8. The shutterless FIR camera ofclaim 1, wherein the IC is further configured to: retrieve predeterminedcalibration tables; apply a shutterless pixel-based correction to theFIR images based on the predetermined calibration tables; and apply ashutterless scene-based noise correction to the FIR images.
 9. Theshutterless FIR camera of claim 8, wherein the calibration tablesinclude at least a gain and an offset value calculated based on twotemperature points for overcoming gain and offset irregularities in theFIR sensor and unifying the FIR sensor's response to infrared (IR)radiation for a range of ambient temperatures.
 10. The shutterless FIRcamera of claim 9, wherein the IC is further configured to: retrieve asubset of calibration values from the calibration tables, wherein thesubset of calibration values is of a specific temperature rangedetermined based on a current measured temperature.
 11. The shutterlessFIR camera of claim 8, wherein the calibration tables include a driftvalue determined for each pixel at each temperature point during acalibration process of the FIR sensor.
 12. The shutterless FIR camera ofclaim 1, further comprising: an image enhancing processor (IEP); whereinthe IEP is further configured to apply correction processes on thecaptured FIR images, wherein the correction processes are applied in apipe-line fashion.
 13. The shutterless FIR camera of claim 12, wherein acorrection process of the correction processes includes: using abilateral filter to preserve edges to reduce noise within the capturedFIR images.
 14. The shutterless FIR camera of claim 12, wherein acorrection process of the correction processes includes: applying anautomatic gain to reduce a distance between a lowest and a highest graylevel within the FIR images, thereby stretching an image histogram ofthe FIR images.
 15. The shutterless FIR camera of claim 12, wherein acorrection process of the correction processes includes: applying acontrast limited adaptive histogram equalization (CLAHE) by dividing theFIR images into overlapping blocks and applying a histogram equalizationwithin each individual block.
 16. The shutterless FIR camera of claim10, wherein a correction process of the correction processes includes: apolarity-and-brightness module configured to perform at least one of:changing white-hot pixels for black-hot pixels and adjusting thebrightness of the FIR images by a predefined offset.
 17. The shutterlessFIR camera of claim 1, wherein the enhanced thermal video stream isconfigured to detect objects in advanced driver assistance systems. 18.The shutterless FIR camera of claim 17, wherein the objects include atleast one of: vehicles, pedestrians, animals, two-wheelers, black-icespots, litter, debris, potholes, gully covers, and curbs.
 19. Theshutterless FIR camera of claim of claim 1, wherein the advanced driverassistance systems include autonomous vehicles systems.
 20. Anelectronic circuit integrated in a shutterless far-infrared (FIR) cameraand configured to process FIR images, comprising: a processingcircuitry; and a memory containing instructions that, when executed bythe processing circuitry, configure the processing circuitry to: performimage corrections, including at least a shutterless correction on theFIR images to output an enhanced thermal video stream.
 21. Theelectronic circuit of claim 20, wherein the electronic circuit and anFIR camera are integrated in advanced driver assistance systems.
 22. Theelectronic circuit of claim 20, wherein the processing circuitry isfurther configured to: retrieve predetermined calibration tables; applya shutterless pixel-based correction to the FIR images based on thepredetermined calibration tables; and apply a shutterless scene-basednoise correction to the FIR images.
 23. The electronic circuit of claim22, wherein the calibration tables include at least a gain and an offsetvalue calculated based on two temperature points for overcoming gain andoffset irregularities in an FIR sensor and unifying the FIR sensor'sresponse to infrared (IR) radiation for a range of ambient temperatures.24. The electronic circuit of claim 22, wherein the processing circuitryis further configured to: retrieve a subset of calibration value fromthe calibration tables, wherein the subset of calibration values is of aspecific temperature range determined based on a current measuredtemperature.
 25. The electronic circuit of claim 22, wherein thecalibration tables include a drift value determined for each pixel ateach temperature point during a calibration process of an FIR sensor.26. The electronic circuit of claim 20, further comprising: an imageenhancing processor (IEP); wherein the IEP is further configured toapply correction processes on the captured FIR images, wherein thecorrection processes are captured in a pipe-line fashion.
 27. Theelectronic circuit of claim 26, wherein a correction process of thecorrection processes includes: using a bilateral filter to preserveedges to reduce noise within the captured FIR images.
 28. The electroniccircuit of claim 26, wherein a correction process of the correctionprocesses includes: applying an automatic gain to reduce a distancebetween a lowest and a highest gray level within the FIR images, therebystretching an image histogram of the FIR images.
 29. The electroniccircuit of claim 26, wherein a correction process of the correctionprocesses includes: applying a contrast limited adaptive histogramequalization (CLAHE) by dividing the FIR images into overlapping blocksand applying a histogram equalization within each individual block. 30.The electronic circuit of claim 26, wherein a correction process of thecorrection processes includes: a polarity-and-brightness moduleconfigured to perform at least one of: changing white-hot pixels forblack-hot pixels and adjusting the brightness of the FIR images by apredefined offset.